Image based method and system for building object model and application states comparison and graphic-based interoperability with an application

ABSTRACT

A system and method for enabling graphic-based interoperability between computer executed applications. A computer system operating as a client may display a graphical user interface (GUI) including control graphic items such as buttons, text boxes, etc. A process may examine the graphical image of the GUI to determine if there has been a change over time in the GUI as displayed which updates a control graphic item. If there has been a change over time in the GUI which updates a control graphic item, an action may be taken, for example updating properties of an object construct corresponding to the control graphic item, raising an event corresponding to an object construct corresponding to the control graphic item, or communicating an event to a process.

RELATED APPLICATION DATA

The present application is a continuation-in-part of prior U.S.application Ser. No. 15/416,484 entitled SYSTEM AND METHOD FOR ENABLINGGRAPHIC-BASED INTEROPERABILITY WITH A RUN-TIME APPLICATION, filed onJan. 26, 2017 and incorporated herein in its entirety.

FIELD OF THE INVENTION

The present invention is in the field of interoperability. Inparticular, the present invention is directed to systems and methods forenabling graphic-based interoperability between computer systems.

BACKGROUND OF THE INVENTION

Interoperability may be the ability of a computer system or applicationto work with other computer systems or applications, typically withoutspecial effort on the part of the users. Although many applicationsexist which can expose to another application or system an externalgraphical user interface (GUI) for interoperability via an applicationprogramming interface (API) or a software development kit (SDK) whichallow interaction with the GUI by low-level programming techniques,there exist applications which cannot be connected or have theirinternal events easily known by any existing technique.

An example of an application which often cannot be connected by standardAPI and/or SDK techniques is an application executing or running in aremote environment (e.g., in a server/client architecture). In thiscase, the user (e.g. using the client system) may be able to see animage of an application on the screen of the client device and performvarious input operations using a keyboard and/or mouse or touchscreen,but existing application integration techniques cannot recognize userinterface (UI) elements or connect to any API exposed by theapplication, even when such APIs exist and are available or ready touse. As such, a third party application attempting to work with oraccess an application may not be able to. Examples of remoteenvironments in which this problem is prevalent include the MicrosoftRemote Desktop system, the Citrix XenApp system, the PCAnywhere system,and the Oracle VM system.

Interoperability problems can be attributed to one or more of forexample lack of a reliable connector (e.g. API or SDK) for aninteraction with such applications; and lack of an object model exposedby GUI elements or objects (for example, buttons, list boxes, links,tables, etc.).

One prior attempt at solving the interoperability issue includestechnology focused on Optical Mark Recognition (OMR), which is used withOptical Character Recognition (OCR) engines to format text whilegenerating a specific text document. OMR also provides the ability torecognize a text document with a template designed by developers.However, OMR and similar technologies do not provide an ability tointeract with an application image, among other deficiencies.

SUMMARY OF THE INVENTION

A system and method for enabling graphic-based interoperability betweencomputer executed applications. A computer system operating as a clientmay display a graphical user interface (GUI) including control graphicitems such as buttons, text boxes, etc. A process may examine thegraphical image of the GUI to determine if there has been a change overtime in the GUI as displayed which updates a control graphic item. Ifthere has been a change over time in the GUI which updates a controlgraphic item, an action may be taken, for example updating properties ofan object construct corresponding to the control graphic item, raisingan event corresponding to an object construct corresponding to thecontrol graphic item, or communicating an event to a process.

Some embodiments of the invention improve the underlying functionalityof the computer systems on which embodiments of the invention areexecuted by for example allowing different executing programs tocommunicate, integrate, operate together, or work together, or to moreefficiently do so. Embodiments of the invention may improve thetechnology of computer application interoperability and communications.For example, embodiments may enable the development and use of run-timesolutions for integration of an application with third partyapplications, such as applications running in remote environments havingonly visual representation of the UI on a client (e.g., a customer)desktop. Application communications independent of platform, operatingsystem or standards, and based only on a visual representation, may beachieved. Some embodiments of the invention enhance functionality ofapplications instantiated on client devices in a server/clientarchitecture, which may make it possible to avoid unnecessaryinstallation, development, or configuration of software on remoteservers and client devices, thus conserving memory and maximizingprocessing power on such servers and devices. Such enhancements mayreduce total cost and development time for applications to be run insuch environments.

Remote servers provided to customers by external information technology(TT) companies may include security policies and/or physical limitations(memory usage, speed requirements, etc.) which do not allow forinstallation of certain applications and/or software components on theseservers. Some embodiments of the invention solve this problem byproviding the functionality of such applications and/or softwarecomponents without requiring actual installation on such servers, and/orrequiring minimal installation on client systems, which may be moreaccessible to a third party than the server controlling the clientapplication. Furthermore, many applications contain components that arenot accessible via existing integration technologies. Some embodimentsof the invention enable access to such components, which may lower timeand effort of research and development to develop connectors to supportthese applications by providing an out-of-the-box solution for it.Embodiments may provide a simple way to adapt or connect twopre-existing software packages. Furthermore, some embodiments of theinvention may function as a universal real-time connector which may beused with any application type. Such a universal real-time connector maybe independent of the application platform (e.g., the desktop, server,etc.) and even independent of any operation system.

These and other aspects, features and advantages will be understood withreference to the following description of certain embodiments of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanied drawings. Embodiments of the invention areillustrated by way of example and not limitation in the figures of theaccompanying drawings, in which like reference numerals indicatecorresponding, analogous or similar elements, and in which:

FIG. 1 is a high-level diagram illustrating an example configuration ofa system for enabling graphic-based interoperability with a run-timeapplication according to at least one embodiment of the invention;

FIG. 2 is a flow diagram of a first part of a method for enablinggraphic-based interoperability with a run-time application according toat least one embodiment of the invention;

FIG. 3 is an example captured application image, according to at leastone embodiment of the invention;

FIG. 4 is an example captured application image shown, with edges andcontours, converted to grayscale, according to at least one embodimentof the invention;

FIG. 5 is an example set of generated bounding shape objects, accordingto at least one embodiment of the invention;

FIG. 6 is an example shapes tree, according to at least one embodimentof the invention;

FIG. 7 is an example scene object model, according to at least oneembodiment of the invention;

FIG. 8 is a scene design-time data table, according to at least oneembodiment of the invention;

FIG. 9 is a flow diagram of a second part of a method for enablinggraphic-based interoperability with a run-time application according toat least one embodiment of the invention;

FIG. 10 is a control methods, properties and events table, according toat least one embodiment of the invention; and

FIG. 11 is a flow diagram a “designer” stage, for identifying and/ordefining GUI objects in an application image during a design-timelearning stage, according to embodiments of the invention; and

FIG. 12 is a flowchart depicting the operation of an RT clientapplication operated side by side with client/server or monitoredapplication on a user terminal, according to one embodiment.

It will be appreciated that, for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn accuratelyor to scale. For example, the dimensions of some of the elements may beexaggerated relative to other elements for clarity, or several physicalcomponents may be included in one functional block or element. Further,where considered appropriate, reference numerals may be repeated amongthe figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Although embodiments of the invention are not limited in this regard,discussions utilizing terms such as, for example, “processing,”“computing,” “calculating,” “determining,” “establishing”, “analyzing”,“checking”, or the like, may refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulates and/or transforms datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information non-transitory processor-readablestorage medium that may store instructions, which when executed by theprocessor, cause the processor to perform operations and/or processes.Although embodiments of the invention are not limited in this regard,the terms “plurality” and “a plurality” as used herein may include, forexample, “multiple” or “two or more”. The terms “plurality” or “aplurality” may be used throughout the specification to describe two ormore components, devices, elements, units, parameters, or the like. Theterm set when used herein may include one or more items. Unlessexplicitly stated, the method embodiments described herein are notconstrained to a particular order or sequence. Additionally, some of thedescribed method embodiments or elements thereof may occur or beperformed simultaneously, at the same point in time, or concurrently.

Some embodiments of the invention enable graphic-based interaction withGUI elements or controls of an application by for example using shapeanalysis or other graphical analysis of a graphical image of the GUIoutput for an application (which may be termed “application image”,which may vary over time as the application state changes), which can beconsidered the display a human user views as produced by theapplication, to identify graphical objects or controls with whichinteraction can be enabled. Some embodiments of the invention provide aGUI object model and/or algorithms for interaction with GUI elements ofan application recognized as components in application image. Someembodiments of the invention recognize well known graphical objects andgraphical object types such as, for example, application windows, GUIcomponents, controls etc., on, e.g., a computer screen image. For thispurpose, some embodiments of the invention apply a contour and/or shapesanalysis of an application image, e.g., from a captured screen image.

Graphical, visual or GUI objects when discussed herein can typicallyinclude visual representations of on-screen graphics or controls, andmay differ from objects in the programming sense, although programmingobjects can represent the visual graphics objects. Objects in theprogramming or software sense may be, in the class-based object-orientedprogramming model, a particular instance of a class and may be calledfor example instantiated objects, or runtime objects. Graphical or GUIobjects may include for example buttons, icons, menus, cursors, textboxes, controls, frame windows, links, list boxes, tables, combination(combo) boxes, tabs, sliders, etc. Graphical or GUI objects may havecorresponding instantiated or software objects. In some embodiments,objects in the programming or software sense and according to theclass-based object-oriented programming model may be not actualinstantiated object-oriented objects, but rather a construct usedinternally by an executing application to represent on-screen objects.

Some embodiments of the invention use for example two phases or modes: adesign-time phase or mode for defining and collecting graphical shapesinformation, and a run-time phase or mode for applying the graphicalshapes information to an application in real time. When referred toherein, design-time may refer to an analysis, preparatory or setup typemode or stage during which embodiments of the invention may analyze anapplication image (which may vary over time as the application statechanges or updates, and which thus may require that the application isexecuted over a period of time), e.g., by applying one or more imageprocessing algorithms, to learn about, identify, define, and/or mark-upobjects in the application image such that the those objects can belater identified, constructed, and/or made operable during a run-timemode or stage. In some embodiments, during a design-time mode, a sceneand/or control anchors (e.g., shape anchors) may be defined, geometricrelationships between anchors may be identified, GUI control types maybe specifying, and controls data, anchors data and/or the applicationimage may be collected and/or stored.

Run-time may refer to mode or stage during which the application isactually running and available for operability. As applied to someembodiments of the invention, during a run-time mode, an appropriatescene (e.g., the application state) may be recognized, for example,based on the collected design-time data, and interaction with GUIelements/objects of application may be enabled, e.g., as though a userwere interacting directly with the application.

In some embodiments, as described in detail herein, during a design-timemode, actions/steps may be implemented such as for example: edges andcontours detection; polygonal approximation of each contour; calculationof shape bounding rectangle(s); defining shape properties (e.g., shapeanchors and/or shape controls), such as approximate points, childrencounts, rectangles, child contours, etc.; defining scene anchors forcontrol shapes. defining geometric relationships between anchors toidentify controls of the scene at run-time; specifying GUI control types(e.g., shape control types), such as frame windows, buttons, links, listboxes, tables, combo boxes, tabs, sliders, etc.; and calculating animage histogram for each shape (e.g., to compare shape histogramsinstead of image comparison). Different and/or other actions may beused.

In some embodiments, during a run-time mode, actions/steps may beimplemented such as for example: recognizing (e.g., identifying,finding, etc.) the frame window shape, e.g., by comparison of the imagehistogram (calculated at design-time) with an image histogram (createdat run-time) or an application image captured at run-time; identifyingone or more shape anchors by comparison of design-time/run-timehistograms and/or the maximum coincidence of child contours; identifyingcontrol shapes by comparison of design-time/run-time histograms andgeometric relationships between control anchors; intercepting imagepaint calls (e.g., BitBlt, StrechBlt, etc.) to identify image changes orupdates (representing, e.g., various application states) and provideperformance optimization for remote control applications; recognizinginput device (e.g. mouse/touchscreen and keyboard) hooks (e.g., via theWindows API) indicating user activity events; recognizing graphicsdevice interface (GDI) drawing functions occurring on the clientcomputer or terminal; recognizing Window or other GUI system events hookand subclass window procedures, e.g., redraw messages; recognizing atimer timeout or the end of a time period; and emulating run-timecontrols and their various actions, properties, and events, etc.Different and/or other actions may be used.

FIG. 1 shows a high level diagram illustrating an example configurationof a system 100 for enabling interoperability with a run-timeapplication, according to at least one embodiment of the invention.System 100 includes network 105, which may include the Internet, one ormore telephony networks, one or more network segments including localarea networks (LAN) and wide area networks (WAN), one or more wirelessnetworks, or a combination thereof. System 100 includes a system server110. In some embodiments, system server 110 may be a stand-alonecomputer system. In other embodiments, system server 110 may include anetwork of operatively connected computing devices, which communicateover network 105. System server 110 may include multiple processingmachines such as computers, and more specifically, stationary devices,mobile devices, terminals, and/or computer servers (collectively,“computing devices”). Communication with these computing devices may be,for example, direct or indirect through further machines that areaccessible to the network 105.

System server 110 may be any suitable computing device and/or dataprocessing apparatus capable of communicating with computing devices,other remote devices or computing networks, receiving, transmitting andstoring electronic information and processing requests as furtherdescribed herein. System server 110 is therefore intended to representvarious forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers and/or networked or cloudbased computing systems capable of employing the systems and methodsdescribed herein.

System server 110 may include a server processor 115 which isoperatively connected to various hardware and software of system 100.Server processor 115 serves to execute instructions to perform variousoperations relating to embodiments of the invention. Server processor115 may be one or a number of computer processors, a central processingunit (CPU), a graphics processing unit (GPU), a multi-processor core, orany other type of processor.

System server 110 may be configured to communicate via communicationinterface 120 with various other devices connected to network 105.Server memory 125 may be accessible by server processor 115, therebyenabling server processor 115 to receive and execute instructions such acode, stored in the memory and/or storage in the form of one or moresoftware modules 130, each module representing one or more code sets.Software modules 130 may include one or more software programs orapplications (collectively referred to as the “server application”)having computer program code or a set of instructions executed partiallyor entirely in server processor 115 for carrying out operations foraspects of the systems and methods disclosed herein, and may be writtenin any combination of one or more programming languages. Serverprocessor 115 may be configured to carry out embodiments of the presentinvention by, for example, executing code or software, and may executethe functionality of the modules as described herein.

Server modules 130 may include more or less actual modules which may beexecuted to enable functionalities of the invention. Server modules 130may be executed entirely on system server 110 as a stand-alone softwarepackage, partly on system server 110 and partly on user device 140, orentirely on user device 140. Device 140 may be remote from third partyserver 180.

Server memory 125 may be, for example, a random access memory (RAM) orany other suitable volatile or non-volatile computer readable storagemedium. Server memory 125 may also include storage which may takevarious forms. For example, the storage may contain one or morecomponents or devices such as a hard drive, a flash memory, a rewritableoptical disk, a rewritable magnetic tape, or some combination of theabove. In addition, the memory and/or storage may be fixed or removable.In addition, memory and/or storage may be local to the system server 110or located remotely.

System server 110 may be connected to one or more database(s) 135, forexample, directly or remotely via network 105. Database 135 may includeany of the memory configurations as described herein, and may be indirect or indirect communication with system server 110. In someembodiments, database 135 may store information relating to userdocuments. In some embodiments, database 135 may store informationrelated to one or more aspects of the invention.

User device 140 may connected to the network 105 and may be any standardcomputing device, for example a desktop computer, smart terminal, dumbterminal, kiosk and/or other machine, each of which generally has one ormore processors, such as user processor 145, configured to execute code,a computer-readable memory, such as user memory 155, a usercommunication interface 150, for connecting to the network 105, one ormore user modules 160, one or more input devices 165, and one or moreoutput devices 170. Typical input devices, such as, for example, inputdevices 165, may include a keyboard, pointing device (e.g., mouse ordigitized stylus), a web-camera, and/or a touch-sensitive display, etc.Typical output devices, such as, for example output device 170 mayinclude one or more of a monitor, display, speaker, printer, etc.

In some embodiments, user module 160 may be executed by user processor145 to provide the various functionalities of user device 140. Inparticular, in some embodiments, user module 160 may provide a userinterface with which a user of user device 140 may interact, to, amongother things, communicate with system server 110. For example, systemserver 110 may execute a client/server, “target” or “monitored”application 142 as the server of a server/client architecture, and userdevice 140 may display the GUI display as controlled by server 110 andaccept input to send to server 110.

In some embodiments, user device 140 may be or act as a “dummy”terminal, by which processing and computing may be performed on systemserver 110, and information may then be provided to user device 140 viaserver communication interface 120 for display and/or basic datamanipulation. In some embodiments, modules depicted as existing onand/or executing on one device may additionally or alternatively existon and/or execute on another device. For example, in some embodiments,one or more modules of server module 130, which is depicted in FIG. 1 asexisting and executing on system server 110, may additionally oralternatively exist and/or execute on user device 140. Likewise, in someembodiments, one or more modules of user module 160, which is depictedin FIG. 1 as existing and executing on user device 140, may additionallyor alternatively exist and/or execute on system server 110.

A computing device discussed herein may be a mobile electronic device(“MED”), which is generally understood in the art as having hardwarecomponents as in the stationary device described above, and beingcapable of embodying the systems and/or methods described herein, butwhich may further include componentry such as wireless communicationscircuitry, etc. Non-limiting examples of typical MEDs are smartphones,personal digital assistants, tablet computers, and the like.

Third party server 180 may operate software or applications which mayinteract with or receive information from target or monitoredapplication 142 operated via system server 110 and displayed on userdevice 140 via methods as disclosed herein. Since third party server 180may not have access to, or API access to, client software operated bysystem server 110 and displayed in a GUI on device 140, embodiments ofthe invention may provide an API for third party server 180 to interactwith the client software via a specially made GUI which gathersinformation from graphical changes on user device 140. For example,system server 110 may execute a client/server, “target” or “monitored”application 142 as the server of a server/client architecture, and userdevice 140 may display the GUI display as controlled by server 110 andaccept input to send to server 110.

For example, third party server 180 may operate software or applicationssuch as RT server software modules 188 (e.g. the Real-Time ProcessOptimization available from NICE, of Raanana, Israel) which are intendedto monitor and possibly control an interaction between a customer and ahuman agent, where the human agent is using application 142: thisinteraction between RT server 188 and agent application 142 may beperformed by embedded or monitoring software typically executed on thesame computer that executes agent application 142 such as RT client 144.Software such as RT server software modules 188 may communicate withagent application 142 and may be remote from agent application 142, andthis communication may be via RT client 144. Third party server 180 mayreceive information from application 142 (which may be considered amonitored application) and in some cases may send messages to or controlapplication 142, for example send messages including suggestions to theagent regarding the customer interaction. One or more items of embeddedsoftware, monitoring software or RT client 144 may be executed by userdevice 140 to monitor and/or communicate with application 142 andgenerate events, alerts, etc., and to operate or act as software objectsto allow interaction between third party server 180 and application 142.Third party server 180 may include one or more input devices and outputdevices such as, for example, keyboards, pointing devices, monitors,displays, speaker, printer, etc. In one embodiment RT server 188 orother software interacts with agent application 142 via embeddedsoftware such as RT client 144. In another embodiment agent application142 is controlled by and sends input to RT client 144 with no otherprogram controlling RT client 144. Generating, triggering or raising anevent may include notifying a process (e.g., a process within or part ofRT client 144, or a process external to RT client 144, such as RT serversoftware modules 188, of the change.

Third party server 180 may communicate over network 105 and may includeone or more processing machines such as computers. Third party server180 may be any suitable computing device and/or data processingapparatus such as servers, laptops, desktops, workstations, personaldigital assistants, etc. Third party server 180 may include one or morecomputer processors 185 which may be configured to carry out methods asdisclosed herein (possibly in conjunction with processors 115 and 145)and may be one or a number of computer processors, a central processingunit (CPU), a graphics processing unit (GPU), a multi-processor core, orany other type of processor. Third party server 180 may include memory182 storing for example data and/or code such as RT server 188. RTserver 188 may include one or more software programs or applicationshaving computer program code or a set of instructions executed partiallyor entirely by processor 185 for carrying out operations as disclosedherein. Modules such as RT server 188 may for example communicate withRT client 144 to receive input from and/or control or send instructionsto monitored application 142, or cause application 142 to receive input,such as text inserted into a textbox, or controls selected or clicked,at the initiative and control of modules such as RT server 188. Suchcommunication may be performed by software objects corresponding tocontrols within application 142. Processor 185 may be configured tocarry out embodiments of the present invention by, for example,executing code or software, and may execute the functionality of RTserver 188 or other modules.

Third party server memory 182 may be, for example, a random accessmemory (RAM) or any other suitable volatile or non-volatile computerreadable storage medium and may include storage which may take variousforms, and may be located remotely.

FIG. 2 is a flow diagram of one embodiment for a “designer” stage, foridentifying and/or defining GUI objects in an application image during adesign-time learning stage, mode, or process, according to embodimentsof the invention. FIG. 2 shows a learning method or process in whichcharacteristic data models or scenes are generated and stored for laterrecall. Furthermore, in some embodiments, method 200 may be configuredto implement one or more of the elements/features/functions of system100.

As with other methods described herein, method 200 may be performed on acomputer having a processor, a memory, and one or more code sets storedin the memory and executed by the processor, such as but not limited tothe devices depicted in FIG. 1. At step 205 when a design-timeapplication image is received. Receiving may include, for example,capturing the image of the application as a “screenshot,” e.g., by useof a print-screen function or other image and/or screen capture methodand/or device, or receiving a previously captured image of theapplication. In some embodiments, the application image may include, forexample, an entire application window, a portion of an applicationwindow, an entire display including portions and/or entire windows ofone or more applications, a portion of a display, etc. For example,turning briefly to FIG. 3, a captured application image 300 is shownaccording to at least one embodiment of the invention.

At step 210, the captured/received application image may be transformedor converted to a greyscale version of the image. For example, the imagemay be processed with a grayscaling function (CvCvtColor) from OpenCVlibrary, or the like. For example, turning briefly to FIG. 4, a capturedapplication image 400 is shown, with edges and contours, converted tograyscale. In some embodiments, transforming or converting theapplication image to grayscale may not be performed, for example, inembodiments where color does not impact detection/identification ofedges and/or contours in the application image and/or in embodimentswhen the application image is already in grayscale.

At step 215, one or more (e.g., two) threshold values may be definedwhich may impact detection of edges and contours in an image. Forexample, in some embodiments of the invention, the processor may receiveone or more threshold values from a user to be implemented in an edgedetection algorithm such as the Canny operator or the Canny EdgeDetector function (CvCanny) from OpenCV library, or the like. In someembodiments, the processor may be configured to automatically determineoptimal or near optimal threshold values for detection of edges.

The Canny edge detector is an edge detection operator that uses amulti-stage algorithm to detect a wide range of edges in images. Cannyedge detection is a technique to extract useful structural informationfrom different visual objects (e.g., objects visually represented in anapplication image) and dramatically reduce the amount of data to beprocessed. In some embodiments, the Canny edge detector may apply aGaussian filter to smooth the image in order to remove noise, find theintensity gradients of the image, apply non-maximum suppression toremove spurious responses to edge detection, track edges by hysteresis,and/or finalize the detection of edges by suppressing edges that areweak and not connected to strong edges, etc.

At step 220, one or more edges and/or one or more contours in theapplication image may be found and/or identified. For example, in someembodiments, the processor may be configured to identify one or moreedges, e.g., by executing an edge detection algorithm such as the CannyEdge Detector function (CvCanny) from OpenCV library, or the like.Furthermore, for example, in some embodiments, the processor may beconfigured to identify one or more contours, e.g., by executing acontour detection algorithm such as the CvFindContours function fromOpenCV library, or the like. Such a contour detection algorithm may findcontours using edges returned from an executed edge detection algorithm(e.g., CvCanny function). Of course, those of ordinary skill in therelevant art will understand that there are a number of algorithms whichmay be implemented, alone or in combination, to filter, identify, and/ordetect edges (Edges detectors), for example: Canny edge detector(operator), Sobel operator, Laplace operator, etc.

In some embodiments, for example when the application image is quitecomplex, visual objects in the application image may be distinguished byidentifying contours (and/or edges) of such visual objects within theapplication image. As understood herein, a contour may be defined as acurve joining a plurality of continuous points (e.g., along theboundary), having the same color or intensity. A contour may be, forexample, an external outline (e.g., stroke) of a visual object thatseparates it from the background and/or other visual objects in theimage. Algorithms known to those of ordinary skill in the art and/or asdescribed herein may be executed in various embodiments to implementconvenient methods for the detection and manipulation of image contours.For example, a FindContours function of the OpenCV library may be usedfor retrieving, detecting, and/or identifying contours. In someembodiments, a processor may implement an approximation method which maycompress one or more horizontal, vertical, and/or diagonal segments,leaving, e.g., only their end points (for example, using aCV_CHAIN_APPROX_SIMPLE method). Detection of edges and contours in anapplication image enables defining of bounding shape objects as resultof this processing, as described herein.

At step 225, one or more shape size values may be defined. A shape sizevalue may define, for example, a minimum and/or maximum perimeter (e.g.,shape size) for which a bounding shape object may be generated. Abounding shape object may be defined as a bounding shape (e.g., arectangle), for example, a minimum bounding shape, surrounding,bounding, enclosing, identifying, and/or otherwise relating to aspecific or given contour, set of contours, set of one or more edgesand/or one or more contours (e.g., typically an identified visual objector other shape), etc. As such, one or more shape size values may bedefined in accordance with embodiments of the invention to define, e.g.,minimum and/or maximum acceptable sizes for bounding shape objects to begenerated as explained herein. Minimum and/or maximum shape values may,for example, prevent the system from bounding unnecessary and/orirrelevant visual objects (e.g., visual objects which are likely toosmall or too large to be GUI control elements or objects of interest tothe user, but which nonetheless have a definable contour). In someembodiments, one or more shape values may be calculated, estimated,recommended, and/or suggested automatically by the processor. In someembodiments, the processor may receive one or more shape values as inputfrom a user. Furthermore, in some embodiments, e.g., when too manyvisual objects are identified, the processor may be configured to removevisual objects/shapes with bounding rectangles less than a previouslydefined shape size, e.g., in real-time, for example, based on feedbackfrom a user.

At step 230 a bounding shape object (e.g., a bounding shape such as aminimum bounding rectangle) for one or more contours, one or more edges,and/or one or more visual objects identified in the application imagemay be created and/or generated. For example, FIG. 5 depicts a set ofgenerated bounding shape objects 500 is shown according to at least oneembodiment of the invention. In should be noted that while in theexample embodiment of FIG. 5 rectangles were generated to bound thevarious objects, in other embodiments other regular and/or irregularshapes may also or alternatively be generated in order to define theboundaries of various visual objects, shapes, contours, and edges withinan application image as appropriate.

At step 235 a shapes tree (e.g., a shape object tree) based on thevarious bounding shape objects generated in step 230 may be built. Insome embodiments, for example, all bounding shape objects with abounding rectangle larger than e.g., a defined minimum rectangle sizemay be placed in a shapes array. This array may be processed, and ashape objects tree may be built. For example, FIG. 6 depicts an exampleportion of a shapes tree 600 according to at least one embodiment of theinvention.

A hierarchy of the shapes tree may be defined using an algorithm based,e.g., on coordinates nesting. Each tree node (e.g., representing abounding shape object) may have an associated set of properties, forexample: path in tree, coordinates (e.g., relative to the top leftcorner of the image), histogram data (as described herein), text if any(e.g., recognized using OCR), child (e.g., internal) shapes andcontours, etc. In some embodiments, the shapes tree may be built inaccordance with nesting coordinates. For example, each shape whichencloses one or more other shapes may be considered as a parent and allshapes enclosed within may be considered as children.

In order to include histogram data in the shapes tree, in someembodiments a histogram (e.g., a design-time histogram) may beconstructed based on the defined bounding shape properties of thevarious bounding shape objects. Histograms can be used to represent suchdiverse information as the color distribution of an object, an edgegradient template of an object, the distribution of probabilitiesrepresenting an expected object location, etc. Therefore, in someembodiments, one or more points of interest may be identified in anapplication image by assigning each point of interest a “tag” consistingof histograms of nearby features. Histograms of edges, colors, cornersand so on may form a general feature type that is passed to classifiersfor object recognition. In some embodiments a first histogram may begenerated during a design-time mode, and a second histogram may begenerated during a run-time mode, at which time the histograms may becompared to enable such object recognition, e.g., based on similarity ofhistogram data.

At steps 240-255, one or more bounding shape properties for one or moreof the bounding shapes (e.g., bounding shape objects) may be defined,identified, associated, received, and/or selected. Such properties mayinclude, for example, defining anchors and controls, etc. At step 240,at least one anchor (e.g., a shape anchor) may be defined, identified,associated, received, and/or selected from among the one or morebounding shape objects. Each anchor may denote shape data selected to beused in the scene identification at run-time. A set of anchors uniquelydetermines a scene. An anchor may be for example an object based on ashape of which internal (e.g., child) content (e.g., contours) isconstant and cannot be changed from design-time to run-time. Forexample, a shape of button with constant text or an image on the face ofthe button may be defined as an anchor as the features of the button arenot expected to change from design-time to run-time and can therefore beused to recognize the application as part of a scene. In someembodiments, one or more anchors may be identified, determined,recommended, and/or suggested automatically by the processor. In someembodiments, a processor may receive one or more anchor selections asinput from a user.

At step 245, at least one control (e.g., a GUI control) may be defined,identified, associated, received, and/or selected from among the one ormore bounding shape objects. A control may associate a given object witha control type for which an interaction with the control is definedwithin the run-time stage. For example, a control type may include oneof a frame window, a button, a link, a list box, a check box, a table, adrop-list, a combination box, a tab, and a slider, etc. In someembodiments, one or more controls may be identified, determined,recommended, and/or suggested automatically by the processor. In someembodiments, the processor may receive one or more control selections asinput from a user. Furthermore, in some embodiments, one or more controltypes for selected controls may be identified, determined, recommended,and/or suggested automatically by the processor. In some embodiments,the processor may receive one or more control type selections as inputfrom a user.

At step 250, one or more bounding shape objects (e.g., additional shapeobjects) may be created on the basis of a shape (e.g., a rectangle)drawn (e.g., manually marked and/or otherwise digitally added) on theimage. Furthermore, the newly created shape object may be added to theshape objects tree.

At step 255, in some embodiments, one or more controls may be attachedor otherwise associated to one or more anchors. By attaching controls todefined anchors, one or more geometric relationships may be definedbetween controls and anchors, the data of which may be used to identifycontrols of a scene at run-time, as described herein. In someembodiments, one or more controls may be attached to one or more anchorsautomatically by a processor. In some embodiments, a processor mayreceive attachment selections as input from a user. Furthermore, in someembodiments, one or more geometric relationships between controls andanchors may be calculated identified, determined, recommended, and/orsuggested automatically. In some embodiments, a processor may receiveone or more geometric relationship calculations or selections as inputfrom a user.

At step 260, data relating to one or more of the design-time applicationimage, the one or more design-time visual objects, the one or moredesign-time bounding shape objects, the one or more bounding shapeproperties, the shape tree data, and/or the design-time histogram datamay be stored, e.g., as a scene or another data object, for laterrecall. A scene may be for example a project object which represents anapplication state. It may include, for example, the captured applicationimage, a set of anchors and/or controls, etc., e.g., defined by a usersuch as a project designer. A scene may provide one or more objectmodels which may be later recalled and used during run-time to identifyan application and enable the various defined controls for theapplication. An object model, as understood herein, may be defined as aset of properties (e.g. data stored by the control), methods (e.g.software functions that an external program may cause the control toexecute), features, and/or events (e.g. software functions may be raisedor fired, or the equivalent of raising or firing may be performedinternal to a program, to produce output to an external program from thecontrol) specific for an identified or given object. FIG. 7 depicts anexample scene object model 700 according an embodiment of the invention.Furthermore, as shown in FIG. 8, various scene data may be provided,recorded, stored, etc., in a scene design-time data table 800 as shownaccording an embodiment of the invention. In some embodiments,additional scenes may be created, e.g., by capturing and processingadditional application images as necessary. Otherwise, the design-timemode may end.

To provide reliable interaction and operability with an application itmay be necessary to identify the application state, particularly openwindows, visible GUI elements, etc. Accordingly, embodiments of theinvention enable a processor to identify the relevant application statebased on the previously stored scene (e.g., scene object model and/orscene table data, etc.). FIG. 9 is a flow diagram a method for enablinginteroperability with a run-time application according to an embodiment.In particular, FIG. 9 depicts a flow diagram of a method 900 foridentifying an application based on detected visual elements (e.g., GUIobjects or control graphic items) of the application during a run-timestage, mode, or process, according to an embodiment. In someembodiments, method 900 may be configured to implement one or more ofthe elements/features/functions of system 100, or another system.

At step 905 a scene (e.g., the scene stored in step 260 of FIG. 2)containing design-time data may be received, recalled, read, and/orreloaded. In some embodiments, while a client application is starting up(e.g., loading), for example, on a remote desktop environment on aclient/user device, the scene may be concurrently recalled and thedesign-time scene data (e.g., design-time (DT) application image,anchors and controls with DT properties, threshold values, shape sizesetc.) may be read and/or loaded. Of course, in other embodiments, scenedata may be preloaded even before an application is run.

At step 910, one or more design-time bounding shape objects forimplementation or use during run-time may be created (e.g., recreated).In some embodiments, one or more of bounding shape objects, acorresponding shapes tree, bounding shape properties, anchors, controls,etc., may be built, generated, constructed, and/or calculated with thesame or similar algorithms, processes, methods, etc., as at design-time,described in detail with regard to FIG. 2. In some embodiments, for eachanchor and/or control defined in the scene during the design-time mode,a sub-image may be extracted from the design-time application image. Insome embodiments, the extraction may be based, for example, on relativecoordinates of the shape object with respect to an anchor or a control.Such sub-images may be used for example, to overlay and/or identifycontrols and/or anchors in an application during run-time, e.g., inplace of bounding objects. In some embodiments, while rebuilding aproject (e.g., a scene and/or one or more components of the scene) andapplying it at run-time, the application image may therefore be splitinto a set of pieces (e.g., saved as sub-images) corresponding to thevarious GUI element recognized. For example, FIG. 7 depicts a List Box,Button Control, Shape Anchor, Frame Window Control, etc., which may beextracted as sub-images, which may be used as overlays during run-timecreation of bounding shape objects.

At step 915, the DT Histogram may be calculated (e.g., recalculated)(e.g., as previously described in step 235). In some embodiments, the DThistogram may be calculated using a sequential call to a library suchas, e.g., OpenCV to execute functions (e.g., CvCreateHist, CvCalcHist,CvNormalizeHist) to create, calculate, and normalize the histogram.

At step 920 a run-time (RT) application image may be captured, and insome embodiments may be saved or cached. Such capturing may be caused orinitiated by any of one or more change triggers, depending on theembodiment, such as a timeout or expiry of a period of time, or adetected screen change or paint command, or a user input that has thepotential to change the screen. As understood herein, capturing mayinclude, for example, capturing the image of the application as a“screenshot,” e.g., by use of a print-screen function or other imageand/or screen capture method and/or device, or receiving a capturedimage of the application, e.g., from an external image capturing device,etc.

Other or different operations may take place. In other embodiments, aprocessor or a process may implement different methods and/or processesto take or capture the application image at run-time, depending on theapplication type and/or the environment in which it is running. Forexample, in some applications (such as the Remote Desktop application,for example) which are drawn on the screen only by image functions, theprocessor may be configured to intercept image paint calls (e.g.,BitBlt, StrechBlt, etc.). For other applications, in some embodiments,the processor may capture a screenshot when an application repaint eventarrives. In some embodiments, the RT application image may include, forexample, an entire application window, a portion of an applicationwindow, an entire display including portions and/or entire windows ofone or more applications, a portion of a display, etc.

At step 925 one or more run-time bounding shape objects (e.g.,instantiated objects corresponding to graphically displayed controls, orinternal representations corresponding to instantiated objects) may becreated, for example, corresponding to, relating to, and/or associatedwith, one or more design-time objects, as described herein. Inparticular, in some embodiments, one or more RT Image bounding shapeobjects and/or a run-time shapes tree may be created, built,constructed, generated, and/or calculated with the same or similaralgorithms, processes, methods, etc., as at design-time, described indetail with regard to FIG. 2, e.g., on the basis of one or moreidentified bounding shape properties as described herein. In someembodiments, the run-time state of the application may not coincideprecisely or at all with the design-time state due to resizing and/orhow the application appears on the screen in other environments, inother applications (windows), etc., and therefore the set of boundingshape objects and/or the shape tree may, in some embodiments, beslightly or substantially different than those generated/created duringdesign-time. As such, in some embodiments, for each anchor and/orcontrol, the processor may be configured to search for appropriateshapes of the suitable size and to detect tree branch overlapped shapes.In some embodiments, one or more (e.g., typically each) of the RTbounding shape objects may be generated and/or created as an appropriatetype so as to expose elements and features of the scene object model,e.g., specific for each object type. In some embodiments, these RTbounding shape objects may be overlaid with corresponding sub-images soas to replicate or otherwise visually represent the correct object typeof the GUI elements for which they were created.

At step 930, in order to identify the proper scene to associate with therun-time application, an attempt may be made to identify the framewindow control of the run-time application, e.g., presuming it wasdefined in the scene at design-time. If no application frame window canbe identified, at step 935, in some embodiments, just the run-timewindow application may be captured, e.g., via a screenshot, e.g., todetect the necessary frame window information. If a frame window controlis identified, then at step 940, one or more frame window shapes in theRT application image may be searched for, detected, found, and/oridentified. In some embodiments, the frame window control in the RTapplication image may be searched for, possibly via its associated data,e.g., by executing one or more histograms comparison functions of DTframe window control data (e.g., collected during design-time) and RTshape histograms (e.g., using the OpenCV function CvCompareHist). Thecomparison result may be the array of RT shape objects whose histogramvalues are similar.

At step 945, one or more RT anchor shapes (e.g., anchors) may be found(e.g., detected, identified). To find RT anchors in the entire scene(e.g., in the scene model and/or in the scene data) or a portionthereof, in some embodiments, a method may again execute one or morehistogram comparison functions, as described above. In some embodiments,a processor may compare design-time object data to image data collectedat run-time, e.g., with or without comparing histograms, depending onthe instance, e.g., by comparing contours. In some embodiments, e.g., ifthe contours are essentially identical, a processor may continueverification of the scene.

Anchors may contain information/data regarding contours calculated forthe shape content (picture, text, lines, etc.) that are typicallyconstant; an anchor may be for example a program or application name, alogo, etc. As such, detection of matching anchors is typically anindication that a correct (matching) scene has been identified. At step950, if no anchors are detected, then, the run-time application imagemay be recaptured at step 955, for example during an application repaintevent or on the occurrence of another change trigger, and the processmay continue with step 925. In some embodiments, recapturing theapplication may resolve any mismatching and/or detection issues.

If one or more anchors are found at step 945, then at step 960, one ormore control shapes (e.g., controls), may be found find (e.g., detected,identified) for example, based on identified anchors. In someembodiments, controls may be searched for in the same or a similarmanner as anchors. To find RT controls in the entire scene (e.g., in thescene model and/or in the scene data) or a portion thereof, in someembodiments one or more histogram comparison functions may be executedagain, as described above. In some embodiments, a processor may comparedesign-time object data to image data collected at run-time, e.g., withor without comparing histograms, depending on the instance. However, itshould be noted that controls are not necessarily constant (as theytypically are with anchors), and their contours may be somewhatdifferent, and thus controls may, in some embodiments, be more readilyidentified via comparison of histograms.

At step 965, if no controls are detected, then the run-time applicationimage may be recaptured at step 955, for example during an applicationrepaint event, and the process continues with step 925. In someembodiments, recapturing the application may resolve any mismatchingand/or detection issues.

At step 970, one or more control shapes (e.g., controls) may be found(e.g., detected, identified, etc.), for example, based on previouslydefined or identified geometric relationships. To find RT controls inthe entire scene (e.g., in the scene model and/or in the scene data) ora portion thereof, in some embodiments, for example, when one or morecontrols have been bound or attached to a particular anchor or anchors,a processor may identify such controls based on searching geometricrelationships between controls and anchors. In some embodiments, such aswhen one or more controls have not been bound or attached to anyspecific anchors, a processor may search for and detect geometricrelationships between controls and any or all anchors already identifiedon the scene. Of course, the processor may execute one or more histogramcomparison functions as well, as described above.

At step 975, if again no controls are detected, the run-time applicationimage may be recaptured at step 955, for example during an applicationrepaint event, and the process may continue with step 925. In someembodiments, recapturing the application may resolve any mismatchingand/or detection issues.

At step 980, in some embodiments any user-drawn controls (e.g., controlshapes) may be found (e.g., detect, identify, etc.). To recognizeuser-drawn control shapes and their respective coordinates, in someembodiments, a processor may calculate geometric relationships betweenthe specific control and any or all other controls and anchors found onthis scene.

At step 985, e.g., when all or a threshold amount of the anchors andcontrols defined in the scene are recognized, the scene may be marked asrecognized (e.g., identified) and interaction with the GUI of therun-time application may be enabled. For example, in some embodiments,in order to realize the object model, a processor may emulate run-timecontrol methods, properties and/or events of the design-time boundingshape objects by, e.g., generating, constructing, assigning, defining,or otherwise enabling the one or more RT bounding shape objects (e.g.,instantiated objects corresponding to graphically displayed controls orinternal constructs using as a model such objects such as objectconstructs) such that they have the same or a similar defined set ofproperties, methods, features, and/or events specific for an identifiedor given object as those of the one or more DT bounding shape objects.For example, for input emulation (mouse, keyboard, etc.) a processor mayuse such functions as SendInput, SendMessage and PostMessage functions(e.g., for the Windows API). The processor may further use mouse andkeyboard hooks (Windows API) to receive user activity events. Forcontent change events a processor may check and identify contourschanging in the control. As before, a processor may also intercept imagepaint calls (BitBlt, StrechBlt etc.) in order to change the applicationstate. Turning briefly to FIG. 10, a table 1000 of example controlmethods, properties, and events is provided according to an embodimentof the invention.

FIG. 11 is a flow diagram of one embodiment for a “designer” stage (e.g.application 146 executed for example by third party server 180 oranother device), for identifying and/or defining GUI objects in anapplication image (e.g. a client/server or monitored application such asapplication 142 executed on a user terminal) during a design-timelearning stage, according to embodiments of the invention. Aspects ofFIG. 2 may be used in conjunction with FIG. 11, and portions of the twoembodiments may be combined. Application 142, shown in FIG. 1 as beingexecuted on a client system, may also be executed on a third partysystem, for the purposes of making its image available to an RT designapplication, and for the purposes of design of RT client 144.

In operation 1100, two processes may be opened or executed on a userterminal such as third party server 180: the application to be analyzed(e.g., application 142 executed during a designer phase), and a RTdesign tool or application 146 operated by a designer to define forexample: graphical objects or controls displayed by application 142 andto be monitored, and software objects which are to represent thegraphical entities (objects, controls, etc.) of application 142. In oneembodiment, the two applications may be operated side by side by thesame processor, but in other embodiments the applications may beexecuted remotely from each other.

In operation 1110 a user may operate application 142 to operate, alteror select a control or other control graphic item displayed by the GUIof application 142, or select another GUI element of application 142.For example, a user may use input devices associated with a computerwhich is part of third party server 180 and click on or select acontrol, text box item, list item, etc. A user may cause a GUI elementof application 142 to be altered, for example by typing text whichappears in and thus changes or updates the appearance of a text box. Auser may essentially operate application 142 to have all controls deemedrelevant by the user operated and thus analyzed and defined, as per theoperations below. A user may highlight or select another GUI element ofapplication 142: for example a user may select (e.g. using a selectionbox or tool) an anchor of application 142.

In operation 1120, with RT design application 146, the control or GUIelement operated or altered in operation 1110 may be recognized, or itsexpected location may be recognized, for example using tools or methodsdiscussed with respect to FIG. 2, or other methods. A display inapplication 146 mirroring the GUI displayed by application 142 mayindicate in highlight (e.g., a colored border) the control or GUIelement operated or altered by the user in application 142.

In operation 1130, with RT design application 146, attributes may beassigned to the highlighted or recognized control operated or altered,possibly with user input. For example, a user may operate application146 to define a recognized control as a text box, button etc., and/ormay define or select a software object (e.g. as shown in FIG. 10) tocorrespond to the recognized control, and a user may define events,methods, and other properties for the software object. While operation1130 may be performed with the aid of user input, all or some ofoperation 1130 may be performed automatically, by a processor.

In operation 1140 RT design application 146 software objects may becreated or defined to be later simulated or instantiated during run time(e.g. as shown in FIG. 10) corresponding to the controls or GUI items,or may create other data corresponding to the GUI items. Such softwareobjects may be object-oriented programming software objects. Dataincluded in or associated with such objects may include for example animage (e.g. a graphical representation) of the control which may be usedas a comparison to a run-time image of the control, and other data. If anon-control element (e.g., anchor) is operated or identified inoperation 1110, it may not be the case that a software object isassigned to the element. For example, the element may be recognized forthe purpose of identifying application 142 or the layout of application142. An anchor may be used during run time for recognizing anapplication, recognizing a position of an application, and defining theexpected other GUI elements or controls, defined by the data created atdesign time as existing spatially relative to the anchor. RT designapplication 146 may also accept user input defining software objects andmay, during the design phase, provide input to controls or otherelements of application 142 and view events or other changes or outputoccurring with respect to software objects associated with thosecontrols.

In operation 1150 RT design application 146 may save the data created,for example as a “scene.” Such data may define or control RT client 144,when it is executed. For example a scene may be compiled into a DLL usedby an RT client application. In some embodiments, data created duringthe design phase may define how RT client 144 operates during run time,or may define or be RT client 144 itself.

Embedded or other software (e.g. embedded software or application suchas RT client 144) may monitor the state of a GUI intensive clientapplication, e.g. application 142, when the application is running. Forexample, RT server 188 may communicate with RT client 144 to receiveinput from and/or control or send instructions to monitored application142, or cause application 142 to receive input, such as text insertedinto a textbox, or controls selected or clicked, via software objectscorresponding to controls within application 142.

Other or different operations may be used.

After design-time, when a client application (e.g. application 142), theapplication may be run at “run time” using the design-time definitionsto allow third-party software to interact with the client applicationwithout direct interaction or access to the server operating the client.In order to do this, during run-time, changes or updates in the GUIdisplay or graphical image of the client application over time may bedetected. The determination of whether there is a change to detect torelevant GUI controls or other GUI aspects of interest to thethird-party software may first be made on the occurrence of a changetrigger, an event or message which may signal the possibility that theclient image has changed in a way relevant to a control or other change.Such a change trigger may be for example one or more of a timeout orexpiry of a period of time, a detected screen change or paint command(e.g. from the OS or Windows level), and a user input that has thepotential to change the screen. Other suitable change triggers may beused.

For example, a first image of the display (e.g. of target, client ormonitored application 142) may be stored or cached (e.g. by RT client144), and at some point, e.g., after a change trigger, a second imagemay be stored or cached, the first image in this context being a priorimage and the second image being the current image. This capturing maybe iterative, where the current image becomes the prior image. Based onrelevant changes or updates (e.g., by detected by RT client 144), aprocess on a client system (e.g. user device 140) communicate changes instate or messages to a third-party system, simulating a GUI. Forexample, events such as TextChanged may fire in response to detectedchanges between a first and second image. “Firing” or “raising” an eventmay in some embodiments be a construct as opposed to an object-orientedraising or firing, and may simply be a process occurring within, or asimulation occurring within, a program such as RT client 144.

For example, after a potential change is detected (e.g. after a changetrigger) the graphical image of the GUI may be examined to determine ifthere has been a change over time in the GUI as displayed which updatesa control or control graphic item (or a monitored control graphic item).A change trigger may cause such an examination or search for changes:for example a timeout or the end of a repeating time period may be achange trigger. A timeout or the end of a time period may be considereda change trigger in and of itself, or in combination with a comparisonof a current screen image to a past or cached screen image which resultsin the detection of a difference between the two images. A changetrigger may include the detection of keyboard input (e.g. on the clientdevice), the detection of mouse input or another indication that the GUI(e.g. on the client device) has changed may trigger a search forchanges.

As a result of this search, actions taken by or on controls or GUIelements may be detected (e.g. a button being clicked, text being addedto a text box by a user or an automatic process). Based on theseactions, simulated instantiated software objects (typically operated bysoftware or application such as RT client 144) may provide output, orthe properties, state or data associated with software objects orsimulated instantiated objects may change (the change typicallycorresponding to the change in the control graphic item): for example anevent may be raised, or fire (typically a simulated event), or the dataor properties reflecting text in a control may change. In someembodiments, software objects are not actually instantiated, and theobjects are an internal representation or fiction that RT client 144uses for itself to access visual on-screen objects; RT client then maycommunicate with a process on a third-party server the text in the GUItextbox displayed by application 142 using such internal (to RT client144) representation. In other embodiments, data for software objects maybe accessible by methods or other devices: for example, a “GetText”method associated with a text box may be called by for example a processon a third-party server, and the GetText method (e.g., operated by orpart of RT client 144) may return the text in the GUI textbox displayedby application 142. Simulating the functionality of the graphicalcontrol may be based on or may be performed after detecting the type ofthe graphical control.

One or more different methods may be used as change triggers or signalsto cause an analysis of an application display for relevant changes tothe GUI or graphical image. It is noted that a change trigger may be inresponse to a change in the application image which does not affect acontrol or GUI or other event of interest, and thus a change trigger maynot result in any event being raised or object properties being changed.Changes may be monitored at the level of the operating system of thecomputer outputting the GUI display. For example, window events hooksand subclass window procedures for handle redraw messages for thedisplay of application 142 may be accessed by RT client 144, and when itis detected, via these messages, that the display of application 142 byuser device 140 (e.g., a graphics card on device 140) has changed, thedisplay may be analyzed to determine which if any controls have changed.This may be performed by a detection of drawing commands altering thevirtual window displayed on device 140 showing the GUI for software 142.Similarly, GDI drawing functions may be intercepted via RT client 144:for example a Redraw event may be raised when the target image or GUI isaltered. GDI intercept may require a native dynamic link library (DLL)injection into the target (e.g. client, such as Citrix) application'smemory space; registering system level breakpoint handlers for all APImethods relevant for windows redraw; using an inter-processcommunication mechanism intended to deliver redraw event information; orthe use of a created object which raises events with the relevantparameters on the occurrence of a redraw event. Input devices 165 ofuser device 140 may be monitored. For example, application 142 maymonitor keyboard and mouse input via hooks, and upon indication that akeystroke or mouse movement or click has been made, the GUI may beanalyzed for changes. A timer or timeout function may cause the GUI tobe analyzed for changes every repeating time period, or if a certainamount of time has passed since the last check for changes.

One or more different methods may be used to determine changes orupdates in a GUI display. Some of those methods are discussed above withrespect to FIG. 9. Further methods are discussed below.

Monitoring of the GUI state and changes in the state may be based onreal time difference calculation of two state application images, forexample a cached, saved or prior image, and a current image. Differencesmay include for example data selection (e.g., background change, borderaround text, etc.) and content data changes or updates (e.g. change intext within a text box). To identify these changes a number of methodsmay be used. In one embodiment, three processes are used in sequence,each providing output to the next: an image common difference regionsidentification process, providing output to a filter background changeregion, in turn providing output to a filter data change regionsprocess. In one embodiment, two different types of image changes may bedetected and used: changes in data selection (e.g., check/uncheck abutton), which can be detected by background or other changes; andchanges in content or data (e.g., changing the content of text).

Applying an image common difference regions identification process(e.g., with mode dilate), GUI images (e.g. one before a change trigger,or an actual or suspected change in the screen, and one current) may becompared using black-white image of absolute differences calculations,such as using the OpenCV Function AbsDiff. A morphology algorithm may beused that smooths pixels shown as changed in the resulting differenceimage (“Difference Image”). The process may use an erode algorithm toobtain region rectangles of changes: rectangular subsets from theoriginal image in which there has been a change. The Difference Imagemay be processed by the OpenCV Function MorphologyEx with the parameterERODE to use an erode algorithm, then on the resulting image, contoursmay be found using for example the OpenCV FindContours Function. Foreach contour a bounding rectangle may be calculated (e.g. OpenCVBoundingRect Function). In one embodiment, rectangles that are below asize threshold, e.g., with width less than two pixels, are removed fromthe rectangle set thus obtained. A number of rectangles, indicatingchanges, or surrounding changed portions, may be thus produced.

The output of image differences processing (a set of rectangles) may befed to background change regions identification, which may identify foreach rectangle of the set of rectangles the type of change: data orcontent changing (which may cause the process to return “false”), asopposed to background changing (which may cause the process to return“true”). This process may recognize changes in black spots: if there areno black spot changes it may be determined that background has notchanged. A process may eliminate the rectangles (input from commondifference regions) without background changes, leaving only rectanglesindicating data changes, in a background change regions identificationprocess. For the remaining rectangles which indicate data changes, theprevious cached image and the current image may be compared with respectto these rectangles: a sub-image pair of rectangles is compared usingfor example a common difference regions method. If any rectangles areoutput, a filter is used to subtract from the sub-images backgroundchanges, to return the data changes in the rectangles.

By creating software objects or control objects, for example havingassociated therewith methods, properties, and events, an embodiment maysimulate an API to a client application. In one embodiment the softwareobjects or control objects are not instantiated and executed, but rathersuch objects are used as a representation within RT client 144, forexample an object construct. Thus various software objects, or controlobjects, may be defined at design time and used internally by RT client144 at runtime. In other embodiments such objects may be actuallyinstantiated. These software objects may be similar to or mimic actualWindows controls APIs. A non-limiting set of GUI visual objects andcorresponding software objects or control objects is described below;other objects may be defined and used.

A generic control may define events, methods and properties which allcontrols may inherit, according to object-oriented methods. For example,a generic control may include the properties or methods of controlcoordinates relative to the application image (e.g. the location of therectangle of the control image); the actual control image (the image ofthe control itself cropped from the application image); certain designtime properties such as the control name, control state, and what kindof control it is, its dimensions; image preprocessing which may preparethe image for OCR, such as digital noise reduction; mouse and keyboardhook settings, which may be hooks to for example the Windows SDK; and ahistogram equalization method that may perform preprocessing to improvethe contrast in an image.

For example, a Button or Link object may be both a visual objectdisplayed on a GUI and a corresponding button or link software object(or object construct used within RT client 144). Each may be similar tothe standard button or link object used in many GUI environments, withsome additional functionality per embodiments of the invention. As withother controls or GUI elements discussed herein, each button or linkobject appearing on a GUI screen may have a separate internal or“instantiated” object within RT client 144 corresponding to the buttonor link, or a software representation or object construct. In someembodiments, each screen object may be represented, in RT client 144, asan object construct including data and description related to theon-screen object; in some embodiments such an object construct can bethought of as similar to an instantiated object-oriented object. Ingeneral, each GUI graphic item of interest may have a separate object(e.g., object construct) corresponding to it executed, instantiated or“imaginary” within RT client 144. In addition, each object or objectconstruct may inherit standard properties, methods and events from ageneric control.

Such a button or link software object may include methods (in the objectoriented programming sense) of for example Click, which may calculatethe control center (the center of the rectangular image of the control)and simulate a left mouse click on the control center in order to setthe focus on the associated button or link with for example theSendInput command; and Get Caption to perform OCR recognition of thecaption or label of the control from the control image and return therecognized text. Such a button or link software object may includeevents (in the object oriented programming sense) of for exampleClicked, fired on a mouse button left up within the control rectangle(the area of mouse click input). Fired events may be processes thatcommunicate occurrences to, e.g. third party software, e.g., RT server188. Properties (in the object oriented programming sense) of suchobjects may include for example, enabled\disabled. Properties may bevalues or data available, via the relevant object, to third partysoftware, e.g., RT server

A process to determine if the object is enabled or disabled and thusdetermine the property enabled\disabled may include for example:creating a ‘gray’ image from the control image (the displayed image ofthe control) with a defined color (e.g. mostly gray, or a uniform grayimage) for each pixel and having the same size as the control image.This gray image may be compared to a gray reference image defined by theuser at design time as having the button or control enabled or disabled.At design time, a ‘rtnorm’ value may be calculated which is the relativedifference norm for the control's design time control image or referenceimage and the defined color image (e.g. the uniform gray image), forexample on a pixel by pixel basis. If the design time reference image isdefined as enabled, and the relative difference norm for the run timecontrol image and the uniform ‘gray’ image (e.g. ‘rtnorm’ value) isgreater than dtnorm, the button or control at run time may be defined asenabled; otherwise the button or control is defined as disabled. If thedesign time reference image is defined as disabled, and the relativedifference norm for the run time control image and the uniform ‘gray’image (e.g. ‘rtnorm’ value) is greater than dtnorm, the button orcontrol at run time may be defined as disabled; otherwise the button orcontrol is defined as enabled.

The relative difference norm may be for example a number distilled froma calculation across all pixels in compared images. Calculating therelative difference norm may be performed using for example:

√{square root over (Σ₁(src1(I)−src2(I))²)}/√{square root over(Σ₁src1(I)²)}

Where a square root is calculated for a sum of each corresponding pixelfor src1 (e.g. the image of the control obtained at design time or runtime) less the corresponding src2 (a uniform gray image) pixel squared,divided by the square root of the sum of each pixel, squared in src1.

For example, a TextBox object may be both a visual object displayed on aGUI and a corresponding TextBox software object. Each may be similar tothe standard TextBox object used in many GUI environments, with someadditional functionality per embodiments of the invention. Such aTextBox software object may include methods of for example Get Text,which may perform OCR recognition of the corresponding text boxdisplayed in the GUI control image (the control image may be thegraphical image displayed on the screen showing the control, a subset ofthe overall image) and return the text; and Set Text, which may allowthe caller (e.g., a process on third party server 180) to insert text inthe corresponding text box on the GUI. For example, processes such as RTserver 188 on a third-party system or RT client 144 may take somecontrol of monitored application 142 by causing text to appear in thedisplay of monitored application 142. RT server 188 may do this usingcalls to software such as RT client 144. Manipulating controls by thethird-party software, e.g. executed on third party server 180, may allowfor example a prompt, instructions, or advice to be given (e.g.,automatically) to an agent operating monitored application 142, mayallow autofill of fields, or other control of monitored application 142.

TextBox software object methods may include for example GetText (obtainthe text in the control using for example OCR); and SetText andAppendText, which may use windows SendInput to set or add text(respectively) to the displayed control textbox. SetText and AppendTextmay use a calculation of the relevant control center and may cause anexecution of a LeftMouse click (set focus) call in order to bring focusto the relevant control, and such methods may use the Windows SendInputcommand (or a similar command, which simulates keyboard typing), andexecution of {CTRL} A, and then {DEL} (select all, and then delete).SendInput or AppendText methods may thus insert or add (respectively)text into the corresponding displayed text box control.

A TextBox software object (e.g. corresponding to a graphical controlallowing input of text) may include events including Text Changed, whichmay fire or be raised when for example RT client 144 detects that textin the text box has changed or updated, notifying for example a processsuch as RT server 188 on third party server 180 that text has changed.This may be fired for example when the previous control state imagestored or cached is found to differ from the current control state imagein a relevant way. Such a change may be detected, for example, bycalculating text changes (when presented with a set of changed areasrectangles) based on relative difference of current control image(typically the graphical image displayed on the screen showing only ormostly the control) with the image stored or cached, using the FilterData Change Regions Method described above. If rectangles not are“empty” (empty indicating no change) then an event is fired.

A CheckBox (e.g., that permits the user to make a binary choice) orRadioButton (e.g., allowing input of a choice of only one of a set ofmutually exclusive options) object may be a visual object displayed on aGUI and a corresponding software object. Each may be similar to thestandard CheckBox or RadioButton objects used in many GUI environments,with some changes according to embodiments of the present invention.Such a software object may include a method of, for example, GetText,which may perform OCR recognition of the corresponding text displayed inthe GUI for the object control image and return the text. Such asoftware object may include a property of, for example,checked/unchecked. Such a software object may include an event of, forexample, a checked/unchecked state change or update, typically fired ona state change or upon the mouse-left-up causing the corresponding statechange, e.g. from checked to unchecked or vice versa.

An algorithm to detect a checked state change (for example for use witha CheckBox or RadioButton) may obtain a bi-level (e.g., binary) image,and apply a threshold function to compare each pixel intensity value athreshold value. Edges detection may use a Laplace operator (e.g., theLaplacian may use the gradient of images). Contours may be identified,and an approximate polygon and rectangle for each contour may becalculated. Then two regions in the CheckBox or RadioButton image may beidentified: the check mark and the text. A check mark area imageidentified at design time (known to have a certain state, e.g., checkedor unchecked) may be matched or compared with such an image updated atruntime.

A Combination Box or ComboBox object (e.g. a box containing acombination of controls, such as sliders, text boxes, and drop-downlists) may also be a visual object and a corresponding software object.Such a software object may include methods of, for example, GetText(returning text produced from OCR recognition); and Select Index (whichmay perform the functions of determining the location of the controlcenter, executing a Left Mouse click (to set the focus, in order tobring focus to the relevant control) with the SendInput command tosimulate a press of the down arrow in the on-screen control n times,where n is the index of selection. Methods may include SelectItem where,for an editable combo box, keyboard input may be sent for control-A(select all) then delete, and then text data and Key “ENTER” may besent, and for a non-editable combo box only text data may be sent bykeyboard simulation, followed by the enter key simulation. Such asoftware object may include events including Selection Changed,returning output that the selection of the object has changed orupdated. This may include finding the location of the combo box itemimage (e.g. using a Detect Item Location algorithm); then calculation oftext changes (where the text change calculation is first presented witha set of changed areas rectangles) based on relative a difference of thecurrent state control image with the image cached, e.g. the prior imagesaved, using for example a Filter Data Change Regions Method describedabove. If the rectangles are not empty (empty rectangles indicating nochange) then an event is fired.

A detect item location algorithm used with a ComboBox object mayinclude, for example, first, obtaining a bi-level (e.g. binary) image. Athreshold function may compare each pixel intensity value with athreshold value. Edges detection may be performed, for example using aLaplace operator (the Laplacian uses the gradient of images). Contouridentification may be performed, and calculation of an approximatepolygon and rectangle for each contour may be performed. Two regions inthe ComboBox image may be identified, for example Button and Text. Theselected item image may be cropped from the text area image of thecontrol.

A ListBox object (a visual object and a corresponding software object)may include methods of, for example, SelectIndex, SelectItem andGetSelectedText. Select Index may perform the functions of calculationof or finding the control center (the center, on the screen, of thecontrol), execution Left Mouse click (to set focus in order to bringfocus to the relevant control), using a SendInput command to simulate apress of the “home” key and the down arrow (the depiction of the downarrow in the on-screen control) control button n times, to move theselection downward, where n is the index of selection. The SelectItemmethod may cause the simulation of an item selected in a list (e.g.,third party server 180 may via RT server 188 cause a selection to besimulated, via RT client 144 in application 142). This may occur forexample by extracting text with their screen or relative coordinates) byfor example OCR, and retrieving the item selected by analyzing the dataof the words extracted. If the item is found (e.g. if the text on thescreen matches the expected text for the item to be selected) then theappropriate mouse click may be simulated (as discussed elsewhere) toselect the item in the relevant application. The GetSelectedText methodmay return the text selected, possibly using a Selected Item Detectionalgorithm (discussed below) or other suitable algorithm. The ItemSelected event may fire if it is determined that the item selected isdifferent than the previously selected item; this may be detected bydetecting changes in the text background color.

The Selected Item Detection algorithm discussed above may include forexample creating a Histogram Equalization of the control image;performing edges detection with for example a Canny edge detector; usinga Hough transform or other suitable technique for finding horizontallines; identifying two regions in the ListBox Image, for example Listbox and Scroll bars areas; processing a selected rectangle withBackground Change Regions Identification Method (discussed elsewhereherein), and finding lines. Then, cropping may be performed on thesub-image of the ListBox Item selected from the list box control image.

In order to control or receive input from any region of the GUI display,beyond those specified by controls, for an application (e.g.,application 142), in one embodiment, any region of the GUI display orapplication image may be considered to be part of a Generic Control,which may have a corresponding software object. A Generic Controlsoftware object may include and fire control actions or events such asClick, Double Click, Get Text and other appropriate events to causeevents on the screen of a monitored application. Such an object mayinclude a set of identification attributes common for all other controlson the screen, as in some embodiments all controls inherit genericproperties, methods and events from a common control. Such a softwareobject may include methods of, for example, Get Text, which returns textobtained by for example OCR recognition from the control image;SendInput, which may create or synthesize keystrokes, mouse motions andclicks; and Scroll, which may for example call a Raise Mouse clickfunction on scrollbars or emulate ‘Page Up/Down’, ‘Up/Down’ keystroke.In order to determine the location of scroll bars and scroll barbuttons, a Scroll Bars Location method or algorithm may be used. Ascroll bar software object may include events of, for example, MouseClicked, which may fire on the occurrence of a Mouse Left Button up in acontrol rectangle.

A “Scroll Bars Location” algorithm, to determine the location of ascroll bar, may be used in conjunction with or as part of a GenericControl. Such an algorithm may include, for example performing ahistogram equalization of the control or GUI image; performing edgedetection using for example a Canny edge detector; using a Houghtransform or other technique for finding horizontal and vertical lines;and identifying scroll bar buttons areas in the GUI or control image,depending on the direction of the scrolling.

At runtime any desktop application may be identified by its framewindow. A visual display on an agent's monitor (e.g. on user device 140)may include a number of windows unrelated to the monitored or clientapplication (e.g., application 142); further, the monitored or clientapplication may appear within a “hosting” window (e.g. a Citrix window)on the agent's monitor. While a window may be found using its “handle”or other known processes using windows, this does not apply for thirdparty software which has no access to the “hosting” window (e.g. aCitrix window) messages. If the monitored application is running inremote desktop environment, its frame windows cannot be recognized byregular window recognition methods. Thus, a monitoring process such asRT client 144 may be able to find the hosting window (surrounding themonitored application image) easily using e.g. its window handle, thetypical API may not be available for the monitored application. Themonitored application's “frame window” may need to be identified withinthe hosting window using its visual, graphical, image. For this purposean object called Frame Control (which can be considered as VirtualWindow) may be instantiated (as with other objects, “instantiated” maybe the equivalent of a virtual construct used in a monitoring process).The frame control object may contain all controls and anchors definedfor the application and may be found or identified by anchors andcontrols contained in it. Anchors may be visual aspects of the clientprogram that do not change, such as the title of the program.

A Frame Window object may represent the monitored application andinclude methods such as Get Caption, which may use OCR to recognize thecaption (e.g. the text area at the top of the application frame, or thetopmost text) of the image. As part of Get Caption, caption and buttonslocations may be searched using for example a Hough Transform techniquefor finding horizontal and vertical lines. A Minimize method maysimulate a left mouse click with the SendInput command on the window'sminimize button to minimize the client window, and a Maximize commandmay simulate the left mouse click with SendInput command on maximizebutton to maximize the window. A Close method may simulate a left mouseclick with the SendInput command on close button. A Bring To Frontmethod may bring the application window to the front if other windowsare over the frame window. A Bring To Front method may determine if theapplication is an icon, and if so the method may set the focus on theicon, then execute or simulate an Alt-Tab keystroke which may show alist of application icons. A screenshot image may be obtained, and theicon location may be found within the image. A mouseclick may then besimulated to maximize the client application. If the Bring To Frontmethod determines the application is not an icon, the design time imagemay be compared to the current frame image, and if the features match, amouse click may be simulated to bring the client application to thefront.

A frame window object may include properties such as icon, which is theApplication image icon, which may be the icon (small visualrepresentation) of the client application when it is minimized, and itsicon image appears on the screen.

A frame window object may include events such as mouse clicked which mayfire event on Mouse Left Button up in the control rectangle.

FIG. 12 is a flow diagram of a method for enabling interoperability oran API with a run-time application according to an embodiment of theinvention. Elements and operations of the embodiment of FIGS. 12 and 9may be used together if appropriate, and may be used with components ofFIG. 1, or another system. In operation 1300, software linking to orcommunicating with a third-party application (e.g., a third partyapplication operated on third party server 180) may be installed on auser device (e.g., user device 140). For example, RT client 144, e.g.embedded software, may be installed and executed on a user device tomonitor and communicate with target or monitored application 142, and toprovide an API from monitored application 142 to RT server 188. Targetor monitored application 142 may be considered a computer executedapplication operated on a computer system, and RT client 144 may also beexecuted on the same computer system to examine the monitoredapplication image or GUI produced by or displayed by the target ormonitored application. In one embodiment, the computer executing aremote, third party program (e.g., RT server software modules 188) andthe computer executing the agent application 142 may be remote from eachother, and the two applications may communicate via RT client 144.

RT client 144 may take as input a scene. In some embodiments, this maybe done by compiling a scene (containing for example informationregarding control images and controls) into a DLL, which may beinstalled on the user device 140 on which an RT client 144 executes, andthe RT client 144 may access the DLL to locate and provide the interfacewith the objects defined by the scene. For example, a native DLL may beinjected into the target application memory space; system levelbreakpoint handlers for all API methods relevant for windows redraw maybe registered on a user device hosting the target application; aninter-process communication mechanism may be created intended to deliverredraw event information to for example software such as RT client 144or to the third party software intended to receive the redrawinformation; and/or an object may be created at design time representingthe relevant API which raises events with the relevant parameters for RTclient 144 or to the third party software intended to receive the redrawinformation. The embedded software may be defined by or controlled bydesign time data, such as a “scene” as discussed elsewhere herein, whichin some embodiments may define the actions of RT client 144 by beingcompiled into a DLL which is used by RT client 144.

In operation 1310, an initial application image, graphical image, or GUIdisplay may be captured, an application image scene may be recognized,e.g. among the various windows on a received image display, the targetclient application image may be found, and an application image,graphical image, or GUI display may be captured and cached or stored forcomparison with an image captured in the future. FIG. 9 depicts anexample embodiment for such application image scene recognition. Suchrecognition may be performed both at start up and also if it isdetermined that an anchor as changed.

In operation 1320 on the occurrence of a change trigger or an indicationthat a screen change significant or relevant to an API may haveoccurred, or in the event that it is determined that an anchor wasupdated, an additional application image, graphical image, or the GUIdisplay for the monitored application is captured and stored or cached.The image captured in operation 1320 may be considered the currentimage, and the image captured previously (e.g. in a previous operation1320 or initially in operation 1310) may be considered the prior image;over time or iterations the current image may become the prior image. Achange trigger may be for example some event or message that indicates ascreen or image change has taken place that might be relevant to the APIor controls on the screen. Such a change trigger can be for example oneor more of the occurrence of the end of a time period or the expirationof a timer, the detection of keyboard, mouse, touchscreen or other inputwhich may cause the change in a screen image, the capture orinterception of an appropriate operating system or graphics systemmessage, or the indication that the GUI or client display has changed.Certain embodiments may use only one of these types of change triggers,and other embodiments may combine more than one of these change trigger,and capture an image on the occurrence of any such change trigger. Otherchange triggers may be used.

For example, a change trigger may be the receipt by RT client 144 of amessage related to a window events hook and subclass window procedurefor handle redraw messages, possibly obtained from the graphics card onthe client computer operating RT client 144. A change trigger may be thereceipt by RT client 144 of a message intercepted from GDI drawingfunctions. A change trigger may be the receipt by RT client 144 of amessage intercepted from monitoring keystrokes and mouseclicks, or otherinputs (e.g., a press on a touch screen) on the target client computer.In other embodiments other processes, such as processes on a third-partyserver, may receive the change triggers.

An application image, graphical image, or the GUI display provided by amonitored application, such as application 142, may be captured orobtained, typically be monitoring software such as RT client 144. Forexample, a screenshot may be obtained. The application image, graphicalimage, GUI display, screenshot or screen image may be saved or cachedfor comparison with a future obtained screenshot or screen image.Typically only the image of the terminal or remote client hosting window(e.g., Citrix) is captured, as opposed to the entire image displayed onthe client computer.

In operation 1330 the graphical image (e.g., the current image) of themonitored GUI may be analyzed or examined (e.g. by comparison with theprior or immediately-in-time prior captured, cached or stored image) todetermine if there has been a change or update over time (e.g. bycomparing the image to a previously cached image). If there is no updatethe process may return to operation 1320 to wait for another trigger orchange event. If there is an update the process may proceed to operation1340.

In operation 1340, if there was a change, it may be determined inoperation 1350 if an anchor was updated or changed. If it is determinedthat an anchor was updated, this may indicate that the change in imagewas for example the client application being moved, minimized,maximized, etc., and the process may move to operation 1320 to capturean image. If it is determined that an anchor was not changed, theprocess may move to operation 1350.

In operation 1350 the graphical image (e.g., the current image) of themonitored GUI may be analyzed or examined (e.g. by comparison with theprior or immediately-in-time prior captured image) to determine if therehas been a change or update over time to controls (e.g. by comparing theimage to a previously cached image), or a change over time that affectsthe API or the relevant monitored controls or other aspects of theclient software, or since the last time the image was examined, in theGUI as displayed. In particular, examination may determine if there wasa change which updates a control graphic item, or a monitored controlgraphic item, e.g. a change in the visual depiction of a graphicalcontrol. This may be performed, for example, by embedded software (e.g.RT client 144) installed on a client device. In some embodiments, whendetermining if a change occurred that affects a control, for eachcontrol, a template image determined at design time may be matched tothe control images during runtime to locate each control.

If there has been no change over time, or if there is no change tocontrol graphic items, or to monitored control graphic items, theprocess may wait for another trigger, e.g. iterate (operation 1320).

If there has been a change over time, or a change or update to controlgraphic items, or to monitored control graphic items, the process may inoperation 1360 take an action with respect to the control graphicitem(s) which have been determined to have changed or updated since thelast image was obtained. An action may include, for example, updating orchanging properties (e.g., object oriented properties accessible byquerying the instantiated object, or simply data associated with aninternal representation (“object construct”) of a screen object) of asoftware object or instantiated object, or an internal representation,associated with or representing the changed control graphic items, orraising or executing an event (e.g., an object-oriented event, afunction which sends a message on certain conditions where the eventsource—e.g., the control software object—sends a message via an event toa listener or event handlers—e.g. the third-party software) or otherwisesending a signal or message corresponding to the software object.Raising an event may include notifying a process (e.g., a portion of RTclient 144, or a process external to RT client 144, such as RT serversoftware modules 188, of the change. For example, an instantiated objector an internal construct (e.g. object construct) associated with thecontrol that has been changed or updated may fire an event or otherwisesend a message. The event may communicate information, such as theoccurrence of a mouseclick or screen touch, a change in state of awindow, or the change to text, to the third party application. Forexample, RT client 144 may communicate state changes of application 142to a third party application executed on third party server 180, vianetwork 105. Ways of communicating the alteration of the properties of avisual object may be used other than object-oriented event or propertiestechniques.

The process may continue by waiting, e.g. iterate (operation 1320), forthe occurrence of a change trigger or an indication that a screen mayhave changed in a way relevant to a GUI.

At any point in the process of FIG. 12, in operation 1370, a third partyapplication may affect the state or provide input to the monitoredapplication. For example, a third party application may send a messageto, call a procedure or execute a process, in order to gatherinformation from or change the state of the monitored application. Thismay be done by an object oriented method call, but may be done by othersoftware methods. For example, an application executing on third partyserver 180 may cause a message to a human agent to appear on the GUIoutput of application 142 via making a method call to a control (e.g. toan instantiated object or an internal construct associated with thecontrol) of RT client 144.

Other or different operations may be used.

Unless explicitly stated, the method embodiments described herein arenot constrained to a particular order or sequence. Furthermore, allformulas described herein are intended as examples only and other ordifferent formulas may be used. Some of the described method embodimentsor elements thereof may occur or be performed at the same point in time.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents may occur to those skilled in the art. It is, therefore, tobe understood that the appended claims are intended to cover all suchmodifications and changes as fall within the true spirit of theinvention.

Various embodiments have been presented. Each of these embodiments mayof course include features from other embodiments presented, andembodiments not specifically described may include various featuresdescribed herein.

1. A method for enabling graphic-based interoperability with a computerexecuted application, the method comprising, on a computer systemoperating as a client and displaying a graphical user interface (GUI),the GUI comprising control graphic items: examining the graphical imageof the GUI to determine if there has been a change over time in the GUIas displayed which updates a control graphic item; if there has been achange over time in the GUI as displayed which updates a control graphicitem, then taking an action, the action selected from: updatingproperties of an object construct corresponding to the control graphicitem; and raising an event corresponding to an object constructcorresponding to the control graphic item.
 2. The method as in claim 1,comprising examining the graphical image of the GUI upon the occurrenceof the end of a time period, the detection of keyboard input, thedetection of mouse input, or the indication that the GUI has changed. 3.The method as in claim 1, wherein the control graphic item comprises oneof a button, a link, a list box, a check box, a table, a drop-list, acombination box, a tab, and a slider.
 4. The method as in claim 1,comprising examining the graphical image of the GUI to identify withinthe GUI a control graphic item.
 5. The method of claim 1, comprisingexecuting the computer executed application and a second program on thecomputer system, the second program to perform the examining operation.6. The method of claim 5, comprising executing server software moduleson a computer remote from the computer system, server software modulesto communicate with the executed application via the second program. 7.The method of claim 1, wherein raising an event comprises determiningthat the state of the control graphic item has changed and notifying aprocess of the change.
 8. A system for enabling graphic-basedinteroperability with a computer executed application, the systemcomprising: a computer system operating as a client comprising a memoryand a processor, the processor configured to: display a graphical userinterface (GUI), the GUI comprising control graphic items; examine thegraphical image of the GUI to determine if there has been a change overtime in the GUI as displayed which updates a control graphic item; ifthere has been a change over time in the GUI as displayed which updatesa control graphic item, then take an action, the action selected from:updating properties of an object construct corresponding to the controlgraphic item; and raising an event corresponding to an object constructcorresponding to the control graphic item.
 9. The system as in claim 8,the processor configured to examine the graphical image of the GUI uponthe occurrence of the end of a time period, the detection of mouseinput, the detection of keyboard input, or the indication that the GUIhas changed.
 10. The system as in claim 8, wherein the control graphicitem comprises one of a button, a link, a list box, a check box, atable, a drop-list, a combination box, a tab, and a slider.
 11. Thesystem as in claim 8, the processor configured to examine the graphicalimage of the GUI to identify within the GUI a control graphic item. 12.The system as in claim 8, the processor configured to execute thecomputer executed application and a second program on the computersystem, the second program to perform the examining operation.
 13. Thesystem as in claim 12, comprising a second computer remote from thecomputer system, the second computer configured to execute serversoftware modules to communicate with the executed application via thesecond program.
 14. The system of claim 8, wherein raising an eventcomprises determining that the state of the control graphic item haschanged and notifying a process of the change.
 15. A method forcommunication between a first executing software program and a secondexecuting software program, the method comprising: the first executingsoftware program obtaining the graphical image displayed by the secondexecuting software program; the first executing software programdetermining if there has been a change in a visual depiction of agraphical control within the obtained graphical image as compared to apreviously cached version of the visual depiction of a graphical controlin a graphical image displayed by the second executing software program;if there has been a change then notifying a process of the change. 16.The method as in claim 15, comprising determining if there has been achange upon the occurrence of the end of a time period, the detection ofkeyboard input, the detection of mouse input, or the indication that theGUI has changed.
 17. The method as in claim 15, comprising simulatingthe functionality of the graphical control based on the type of thegraphical control.
 18. The method as in claim 15, comprising examiningthe obtained graphical image to identify within the obtained graphicalimage a graphical control.
 19. The method of claim 15, comprisingexecuting the first executing software program and the second executingsoftware program on the same computer.
 20. The method of claim 19,comprising executing server software modules on a second computer remotefrom the computer, the server software modules to communicate with thesecond executing software program via the first executing program.